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ABSTRACT 



This booklet discusses the following critical issues in 
evaluating the effectiveness of technology in education: (1) The 

effectiveness of technology is embedded in the effectiveness of other school 
improvement efforts; (2) Current practices for evaluating the impact of 
technology in education need broadening; (3) Standardized test scores offer 
limited formative information with which to drive the development of a 
school's technology program; (4) Schools must document and report their 
evaluation findings in ways that satisfy diverse stakeholders' need to know; 
(5) In order for evaluation efforts to provide stakeholders with answers to 
their questions about the effectiveness of technology in education, everyone 
must agree on a common language and standards of practice for measuring how 
schools achieve that end; (6) The role of teachers is crucial in evaluating 
the effectiveness of technology in schools, but the burden of proof is not 
solely theirs; and (7) Implementing an innovation in schools can result in 
practice running before policy. The accompanying CD-ROM presents a three- fold 
approach to designing an evaluation for technology education that asks school 
improvement teams to think about evaluating technology from the systemic 
organization, teaching practice, and student learning perspectives. (MES) 
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"We are far enough along in 
the technological revolution and its 
application to learning that it is time 
for systematic review and analysis 
of what works best-" 

U.S. Secretary of Education Richard W. Riley 

Secretary's Conference on Educational Technology 
Washington, D.C., July 12, 1999 
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The Secretary's Conference on Educational Technology: Evaluating the 
Effectiveness of Technology on July 12-13, 1999, in Washington, D.C., noted 
a shift in schools' focus on technology. Where once the emphasis was on 
building and implementing a technology infrastructure, today it is on evaluating 
the effectiveness of its use in schools and classrooms. Parents and teachers, 
school boards and administrators, governors and state legislatures, and 
Congress all want to know if the nation's investment in technology is providing 
a return in student achievement. Indeed, if resources are to be expended 
on technology, it is becoming a political, economic, and public policy necessity 
to demonstrate its vital effectiveness. 

In his opening address, U.S. Secretary of Education Richard Riley remarked, 
"The primary reason for this conference is to gather information from all of 
the outstanding schools, districts, and states represented here — so that we can 
study it, share it, and learn from it. Just as important as learning what works, 
we must learn what does not work. We must not assume everything that 
employs technology is going to be successful. That is why evaluation is so 
important. And then we must use that evaluation to create positive change." 

The conference drew on the insights and collective wisdom of its attendees, 
starting with an emphasis on state-level technology evaluations. Evaluators 
from West Virginia explained how they isolated the effects of their Basic 
Skills/Computer Education initiative. They found that the more access to 
technology students had and the more their teachers believed that technology 
could help and were trained to use the technology, the higher students scored 
on the Stanford 9 (11% of the total gain scores). Idaho attendees described a 
four-year study focused on eight specific goals by which to evaluate the impact 
of the state's technology investment. Significant results included statewide 
academic gains as measured by the Iowa Test of Basic Skills (ITBS) and 
the Test of Academic Proficiency (TAP) for 8th and 11th graders. 
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From there, the conference turned its spotlight on school practitioners who 
expressed the need for more formative evaluations than the summative 
evaluations described by state policymakers. Lively debates arose among 
teachers, district curriculum and technology coordinators, administrators, 
state curriculum and technology coordinators, state and national policymakers, 
and researchers in the evaluation of educational technology about ways of 
identifying and collecting technology evaluation data that is relevant at 
the local level but also useful for other stakeholders. 

During what Dale Mann of Interactive, Inc., called, "this developmental moment," 
conference participants exchanged promising evaluation strategies and 
techniques and considered how to respond to the many voices demanding to 
know technology's effects on schooling. The following seven critical issues in 
evaluating the effectiveness of technology in education arose as a consequence 
of the interaction among stakeholders: 

p> The effectiveness of technology is embedded in the effectiveness of 
other school improvement efforts. 

[> Current practices for evaluating the impact of technology in education 
need broadening. 

j> Standardized test scores offer limited formative information with 
which to drive the development of a school's technology program. 

Most schools are looking for additional means for collecting useful 
data for this purpose. 

[;> Schools must document and report their evaluation findings in ways 
that satisfy diverse stakeholders' need to know. 

P> In order for evaluation efforts to provide stakeholders with answers 
to their questions about the effectiveness of technology in education, 
everyone must agree on a common language and standards of practice 
for measuring how schools achieve that end. 

p> The role of teachers is crucial in evaluating the effectiveness of technology 
in schools, but the burden of proof is not solely theirs. 

iv x 

Implementing an innovation in schools can result in practice running 
before policy. Some existing policies need to be "transformed" to match 
the new needs of schools using technology. 
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The effectiveness of technology is embedded] in i 
the effectiveness of other school improvemeoiit efforts, j 
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"Linking technology with core instructional j 
objectives is what makes goodn effective 
use of technology" That's the message we 
need to communicate- It's a process — 
not a number-'' 



Margaret Honey 

Center for Children and Technology 



A school's vision of the education it strives to provide students 
contains many elements, of which technology is but one. Other 
elements in this vision include administrative procedures, curricula, 
classroom organization, teachers' pedagogical approaches, time 
and space designations, school-community partnerships, and logistical 
and social factors. Developing ways to isolate the effects of technology 
within a dynamic environment where so many elements work together 
is one of evaluation’s most challenging issues. 

Evaluators at the conference argued that social phenomena such 
as learning contain so many interacting factors that traditional 
experimental designs don't yield effective information. They support 
using evaluation designs that penetrate the effects of implementing 
technology at both individual, organizational, and sometimes even 
community levels. Evaluation designs of this type may be based on 
a system of learning benchmarks and other new forms of assessments 
that take the "localness" of evaluation into account. 

The high-stake decisions linked to technology implementation 
pressure educators to demonstrate that technology makes a difference 
in student learning. Many educators fear that evaluation places 
their technology programs at risk if they cannot produce measurable 
results in a relatively short time. The message that needs to be 
conveyed about the effectiveness of technology is that implementation 
of any sort produces outcomes. These outcomes, however, will be 
different at different stages of implementation. 
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At Mantua Elementary School in Virginia, technology is viewed not as 
an end in itself, but rather as a tool that augments the following four 
pillars of the Basic School, which are (1) the School as Community 
(bringing into focus how people relate to one another and work 
cooperatively to solve problems); (2) a Curriculum with Coherence 
(bringing an interdisciplinary approach to the acquisition of knowledge); 

(3) a Climate for Learning (providing the physical and motivational 
factors necessary for effective teaching and learning); and (4) a 
Commitment to Character (emphasizing how the school experience 
shapes the ethical and moral lives of children). The school community 
uses technology to simplify, facilitate, and enhance individualized and 
social learning processes within its interdisciplinary curriculum. 

Teachers are seen as leaders, facilitators, and mentors, well grounded 
in technology implementation strategies and well trained in the use 
of the most current computing equipment and software applications. 
Children exposed to interdisciplinary units of study use technology 
as a tool to become literate, cooperative, problem-solving, self-motivated . 
learners and that is what Mantua is all about. What most distinguishes 
education at Mantua Elementary is that its students are not passive 
recipients of knowledge, but rather, active participants in the full 
educational process. 

A technology-rich environment can support initiatives focused on 
improving learning outcomes as shown in Union City, New Jersey. 

The district framed its technology evaluation in conjunction with 
evaluation of school reforms such as students' development of literacy, 
higher-order thinking, and collaboration skills. With district funds and funds 
from Bell Atlantic and the National Science Foundation, technology became 
a key catalyst for school improvements that led to measurable academic 
achievements. But, as one of the most impoverished urban communities 
in the United States, Union City faced an uphill battle against state takeover 
with more than a plan to implement technology. Technology was just one 
of eight key reform strategies integral to the district's reported success 
in making school improvements. 
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i Current practices for evaluating the impact 

of technology in education need broadening. | 
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"To a certain extenti we are living out 
the decisions reflected in previous evaluation 
methods! which constrain our thinking about 
the purpose and effectiveness of technology 
in education"" 

i 

Walter Heinecke 

Curry School of Education at the University of Virginia , 



The issue that confronts schools is broader than technology. It is 
about learning and the need to find new ways to identify and measure 
the skills and knowledge that students gain from using technology. 

It is about stakeholders' needs for information beyond self-report 
analyses and traditional standardized testing. It is about building the 
capacity of teachers to evaluate technology resources and to align 
their uses with the learning goals and content standards of the 
curriculum. It is about evaluating technology implementation efforts, 
curriculum integration methods, and learning processes in order to 
make sound decisions for continual improvement. Ultimately, the 
issue is about involving the key stakeholders, identifying appropriate 
measurable indicators, and developing reliable instruments that will 
yield insightful and valid information about what makes educational 
technology effective. 

The multimedia and networked capacities of the technology 
infrastructure are radically altering the face of technology-related 
practices in schools. However, the same rich diversity of technological 
tools that has created new learning opportunities for students has 
complicated the standardization of technology assessment. The fact 
that technology tools have undergone rapid cycles of innovation, 
causing constant change in the types of evaluation questions that 
need to be asked, compounds the difficulty even more. Educators, 
evaluators, and developers of measurement instruments struggle 
to keep current with the rush of information needs having to do 
with technology's effectiveness. 
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In order to understand changing evaluation practices, stakeholders from the 
policy level on down to the home need information on how using technology 
changes teaching and learning, its organizational impact, and the outcomes that 
can be reasonably expected at different stages of technology's implementation. 

In short, the challenge facing educators and evaluators is to compile enough 
evaluation data to substantiate and articulate technology's place in student 
and teacher learning. 

\s> Many schools at the conference reported that they do use standardized 
tests as a part of their technology assessment program, but they also look 
at other outcomes. Educators in the Cherry Creek School District in 
Colorado, for instance, are using methods they call "far from refined." 

They evaluate progress based on district goals such as developing students' 
higher-order thinking skills, promoting collaboration among students 
working on projects, and honing the research skills of students around 
real-world topics. Instead of conducting quantitative research, they rely 
on best practices uncovered by currently published research to guide 
their technology implementation. The district's philosophy regarding 
evaluation is that isolating technology as the cause of achievement, 
productivity, or change is impossible. Therefore, they evaluate systemically: 
looking at SAT and ITBS scores related to programs in which technology 
is used; analyzing results from their Technology Integration/Student 
Achievement Specialist Survey; using the National Educational Technology 
Standards to develop ways of measuring student progress in technology 
foundational skills — to name a few of their multiple measures. 

Technology "ubiquity" in supporting other programs has convinced skeptics 
of its value. Schools at the conference suggested that were it not for the 
access to people, resources, and ideas that technology provides, school 
programs, from student peer-mentoring and summer enrichment to 
teacher professional development, would be seriously crippled. 
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Standardized test scores offer limited formative 
information with which to drive the development 
of a school's technology program. iVlost schools 
are looking for additional means for collecting 
useful data for this purppsd,. 
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"Who gave legislators reading and math test 
score to begin with? We did- We need to give 
them other measures! tell them how technology 
works! and help them see the results-" 

David Dwyer 
On-Track Learning, Inc. 
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Standardized tests scores have become the accepted measure with 
which policymakers and the public gauge the benefits of educational 
investments. But educators and evaluation researchers argue that these; 
scores say little about how to improve technology's effectiveness in 
schools. For this, they need information from formative evaluation. 
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Formative evaluation tells what technology applications work, under 
what conditions, and with which students. It supplies information on 
how technology affects student attitudes toward learning. It can show 
the impact of technology on promoting collaboration among diverse 
learners. It can track technology literacy skills development and indicate 
the impact of technology access. Formative evaluation can tell teachers 
about their students' progress toward developing the skills to access, 
explore, and integrate information; think at high levels; and design, 
experiment, and model complex phenomena. 
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Formative evaluation also yields information on the effectiveness of professional 
development activities, the adequacy of school management systems, and 
other issues having to do with building the school technology infrastructure. 



The good news is that schools have access to more information on these 
questions than they might think. Evidence of technology effectiveness may lie 
in fewer disciplinary referrals, students' completing more complex homework 
assignments, a new robustness in student performances, students taking more 
difficult electives or requesting particular teachers and courses, increases in 
requests for equipment and technical assistance, declines in special education 
placements, lower drop-out rates, rises in college applications and acceptances, 
increases in student job offers, and more parent participation. 

Other information collected through simple observations and questionnaires 
is formative as well. What technologies do teachers and students use and why? 
What is their attitude toward them? How has technology changed how teachers 
teach? How has it affected students' engagement with learning materials? 

Even the use of physical space, such as the rearrangement of study carrels 
in spaces where students can engage in learning with their peers, for example, 
can symbolize changes brought on by the use of technology. 

The problem is not so much the lack of data. The controversy revolves around 
accountability measures that ask the right evaluation questions; identify appro- 
priate data sources; systemically capture the data; and analyze, interpret, and 
report the data in its appropriate context. 

j^> An educator from East Brunswick Public Schools in New Jersey maintained 
that of all the ways to evaluate technology integration, including hiring 
external consultants to conduct an evaluation, "the easiest to look at is 
standardized test scores." The most compelling evidence, however, is in 
what the district calls "secondary indicators." According to one of these 
indicators, when technology was integrated into a ninth-grade science 
curriculum, enrollment in chemistry classes swelled by nearly 500 percent, 
with overall enrollment in science courses growing by 17 percent. 




Other educators at the conference reported discovering innovative 
indicators with which to evaluate technology's effectiveness: 

High school humanities teachers in Oswego, New York, noted more 
varied citations in student papers after students began doing their 
research on the World Wide Web. 

The technology director in Montgomery, Alabama, observed that 
teachers put more detail and illustrative resources into their lesson 
plans than they used to do. 

Educators in Iowa used Bloom's taxonomy of cognitive learning 
as a guide to observing technology-integrated learning units. 

They found that technology-integrated learning reached higher 
in Bloom's hierarchy than non-technology-integrated learning. 

A count in several districts showed that interdisciplinary instruction 
was more prevalent in technology-supported instruction. 

A technology coordinator from Anderson County Schools, Tennessee, 
summed up technology's effectiveness this way: "I know there is 
great impact because if the file server drops, teachers want to call 
the buses and go home." 

When test scores in Blackfoot School District, Idaho, revealed that 
students who used technology in their coursework scored 15 percent 
higher than those who did not use technology, no one in the 
community questioned technology's role or the capital investments 
that the district had made. Yet, when officials couldn't pinpoint 
a more exact effect of technology on student learning, they knew 
that their evaluation of technology effectiveness had to go deeper. 

They analyzed each piece of their technology system, including 
the role of learning benchmarks in content areas and grade levels, 
the usefulness of professional development activities, the unique 
effects of particular software, and the nature and goal of instructional 
activities. 




P> Reporting students' achievement of core competencies on network 
technology has provoked new interest in school improvement 
in several communities. In almost every case, posting these 
competencies sparked districtwide debate about the relevance 
of present standards. These dialogues drove districts' examination 
of student achievement deeper than ever before, resulting in 
teachers being better informed and more committed to addressing 
agreed-upon competencies. 
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Schools must document and report their evaluation j 



Interest in the effectiveness of technology is at an all-time high. 

Parents want to know if children are developing a sound content base 
and thinking skills, and if they are going to be capable of lifelong 
learning in a fast-paced technological society. Teachers want to know 
if technology tools will help facilitate what they want to happen in the 
classroom. Administrators want to know if professional development 
activities are improving the way teachers use technology to teach. 
Funders, policymakers, and taxpayers want to know if technology 
is sufficiently promising to continue investing in it. Documenting 
and reporting evaluation data to meet these diverse stakeholders' 
need to know presents educational evaluators with a daunting 
series of challenges. 

The difference in the data needs of policymakers and educators is 
particularly acute. While policymakers want to see data on the isolated 
effects of technology, educators need information that is tied to 
systemic practices. Policymakers tend to value summative reports 
documenting student achievement, while teachers and administrators 
value formative reports documenting implementation outcomes in 
order to make sound decisions about their technology plans. Many 
kinds of data are important, but each fails to satisfy the other. The 
best hope of closing this gap lies in helping all stakeholders see 
(1) the importance of technology as an effective component of the 
educational system, (2) how technology is and isn't capable of making 
a difference in curriculum and instruction, and (3) how innovative 
practices of teaching and learning with technology require multiple 
measures in order to verify the impact. 



findings in ways that satisfy diverse stakeholders' 
need to knoyy. j j ;i ;! 
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"We cannot survive on the random 
story anymore-" 



Linda Roberts 

Office of Educational Technology 
at the U.S. Department of Education 




I 1 



ERIC 



BEST COPY AVAILABLE 



16 




Comparative language speaks loudly in this regard. It is useful to show 
technology's effects in a tangible way by, for example, comparing the instruction- 
al practices and learning opportunities that students have with technology 
to instruction without technology. Open dialogue and an understanding of 
mutual expectations for performance throughout the technology implementation 
process can resolve much of what differentiates stakeholders' interests in 
technology outcomes. What information do these groups need? What type and 
how much documentation do they require? What standards of documentation 
are most useful to different stakeholders? These are useful questions to consider. 



Finally, communicating about evaluation requires "speaking to" the stakeholder 
audience. What is the audience's level of technology sophistication? How 
knowledgeable is the audience about evaluation terms and procedures? 

Speaking the language of the audience — by converting effect sizes into months 
of academic gain, for example — influences the way people think about technology 
and their support for it. The technology infrastructure, itself, can be a useful tool 
for capturing, interpreting, and reporting data from multiple measures into 
understandable terms for a variety of stakeholder audiences. 




When schools in the Juneau, Alaska, instituted electronic report cards 
to inform parents how and in what ways their children were meeting 
core content standards, communications between teachers and parents 
surged. A middle school teacher described this new type of access to 
parents as "empowering" her partnership with parents to guide their 
children's learning. 




Educators at the conference considered parents one of their most 
important audiences. Once parents understood the value of technology, 
they became advocates. Parents, in fact, were often instrumental in 
moving technology into the classroom. The question that lingers is how 
to spread the message from parents to legislators. Some part of the 
answer, conference goers maintained, lies in encouraging parents to 
bear the message to policymaking bodies. 



Conferees voiced concerns about the media's portrayal of technology 
programs in many of the nation's schools and districts. While applauding 
the media's role in informing the public about technology, educators 
charged that its interest in profiling technology growth and use "in 
one chart on one page" shortchanges the diversity of outcomes that 
technology produces. Participants suggested that the best way to 
encourage more comprehensive portrayals of school technology 
programs in the media is to link technology outcomes to goals that 
are deeply embedded in the mission and culture of the school. 





In order for evaluation efforts to provide stakeholders 
with answers to their questions about the effective- 
ness of technology in education, everyone must agree 
on a common language and standards of practice 
for measuring how schools achieve that end. 

i 

"You have to show people the qualitative 
difference in what kids can actually do-" j 



Eva Baker 

National Center for Research on Evaluation, 
Standards and Student Testing at the 
University of California, Los Angeles 



Dialog among stakeholders plays a central role in evaluation efforts. 
Stakeholders must be attuned to common goals for the uses of 
technology, information needs, cultural terms, and methods for 
measuring outcomes. They must have consensus around roles 
and a clear vision of where they are going and the steps they 
need to take to get there. 

State-level consortia, made up of representatives from many stake- 
holder groups, can help develop guidelines that address schools' 
questions such as: What are important technology-induced indicators 
in our state and what instruments are available to measure these 
indicators? Where are the gaps in evaluation needs and measurement 
tools within our school communities? How can district educators 
and university researchers collaborate to develop evaluation instru- 
ments that will measure technology's effectiveness in our schools? 

Educators have known for a long time that technology can help 
students learn basic skills. But the tools that measure basic skills 
don't evaluate how technology supports students in developing 
capacities to think creatively and critically and vice versa. There 
is a need to develop additional evaluation tools that can help 
measure whether students are learning the "new basics" such 
as computer literacy, collaborative teamwork skills, and lifelong 
learning abilities. 
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Left to themselves, schools have little time to develop and test such evaluation 
tools. While the successful evaluation of a school's technology does not 
necessarily require that researchers and evaluators be on the scene, seeking 
such expertise can be helpful, especially in evaluations that encompass several 
buildings or districts. Many universities offer technology evaluation expertise. 

In addition, regional educational laboratories and technology education consortia 
allocate many of their resources to helping schools address evaluation issues. 
Other for-profit and not-for-profit organizations can also be helpful. Still, it is 
difficult for schools to identify what assistance is available. The field is ripe 
for developing scalable approaches, tools, and strategies for evaluating 
the effectiveness of educational technologies. 

The most useful tools yield information that is specific to the given student 
population and that allows teachers to track students' progress over time. 

Tools also need to measure those aspects and outcomes of learning that 
would otherwise be unattainable without the use of technology. Evaluation 
that demonstrates what students can do with technology that they couldn't do 
before access to the technology shows impact. For example, performance 
measures — observations of what students do and where they go on the Internet 
and how students collaborate with each other — help teachers track the impact 
of technology on student learning. Other measures that tap into education's 
broader curriculum aims include projects, essays, and extended performances. 

Several school district representatives reported replacing student 
technology competency requirements with technology/content area 
integration standards as a basis for benchmarking grade-level technology 
integration. Their rationale was that this shift emphasizes technology's 
supportive role in teaching and learning rather than making technology 
use an end in itself. These educators believe that indicators articulating 
the components of a model instructional unit in fourth-grade science, 
for example, are more useful than technology competencies students 
should demonstrate at the fourth-grade level. 

p> Early in the conference, it became clear that technology has spurred new 
terms and/or word meanings in our vocabulary. The term "engaged 
learning," for instance, had a very different meaning for participants from 
Chicago Public Schools than it did for educators in Cherokee County School 
District in Alabama. Similarly, when one participant referred to "perform- 
ance standards," educators from New Hampshire's Campton School District 
envisioned a very different set of standards than did their colleagues from 
the Okaloosa County School District in Florida. These exchanges illustrated 
the need to come to consensus on the terms and language of the 
evaluation process. Terms such as "technology integration," "benchmark," 
"core competency," "alternative assessment," and even "evaluation" and 
"student achievement" elicit different meanings from a range of educators 
and, unless they are made clear, can undermine evaluation efforts. 

19 
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"Evaluation is part of a reflective process- 
The more reflective we arei the more likely I 
we are to improve our practice-" j 

Charol Shakeshafi 

j 

Hofstra University' 



Technology has revolutionalized what teachers do. It has added 
new breadth and depth to instruction. This, in turn, has transformed 
the role of the classroom teacher. In reformed educational settings, 
teachers guide students in using telecommunications to interact 
with astronauts in space, searching the Internet for up-to-the-minute 
information, and programming technology systems to help solve 
local or global problems. 

The countless hours teachers spend observing and interacting 
with students make teachers a rich source of data about the impact 
technology has on student learning. Teachers are the first to recognize 
increases in students' self-esteem and confidence, enhanced content 
area understanding, and more informed and empathic responses to 
world events as a result of using technology. This new role for teachers 
underscores the need for high-quality professional development in 
the use of technology and in determining what and how students 
learn best with technology tools. 

What teachers know about their students and about technology 
determines their competence in day-to day classroom decision making. 
Good teachers evaluate their students and make decisions about how 
technology can boost their learning on a daily basis. Do students 
have access to the appropriate technology resources and tools? 

Are students using the technology efficiently? What kinds of learning 
tasks will challenge students' creative and critical thinking? In this new 
technology environment where there is not one instructional strategy 
but many, teachers need to know how to manage interactive group 
dynamics as well as technological systems. 
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Professional development in schools that have implemented and evaluated 
educational technologies successfully helps teachers link effective uses of 
technology to impacts on student learning. Evidence of technology literacy, 
faculty meeting agendas, lesson plans, and classroom observations are all 
ways to determine a teacher's grasp of technology as a learning tool. The 
most useful program evaluation is one in which a strong formative element 
examines the connection between instructional practice, technology uses, 
and learning outcomes. 



Teachers are integral to the process of evaluating technology initiatives. They 
can act as partners with researchers to identify the sometimes very subtle 
effects associated with technology uses. Teachers can also play key roles in 
measuring and documenting changes in student learning as they occur. Some 
of the best results in evaluating technology come from schools recognizing 
and harnessing the expertise teachers have in identifying technology-induced 
learning outcomes. 



Teachers who have learned to use technology effectively in the classroom are 
convincing their colleagues of technology's potential. Teachers training teachers 
to evaluate the usefulness of technology in the classroom remains a potent 
professional development strategy. 

p> Evaluators must learn to trust teachers' ability to determine and describe 
technology's "ripple" effects, Margaret Honey, director of the Center for 
Children and Technology, explained during the conference. Success in 
studying school technology programs, according to Dr. Honey, often 
hinges on teachers contributing to the development of research questions 
and sharing ideas on howto record key indicators of effectiveness. 




Lennox School District in southern California builds teachers' capacity 
to evaluate student learning with technology by having teachers 
collaboratively score students' work. Examining student products together 
builds consensus among teachers on the curriculum's core goals and 
the types of assessments that measure the achievement of those goals. 




The nature of the questions that teachers ask their technology coordinators 
can serve as data for evaluation. Their questions can indicate a school's 
position along the continuum of technology implementation. A technology 
coordinator from Helena, Montana, observed that when schools first 
deployed technology, teachers' questions centered on getting the hardware 
to work. Only a couple of years later, these same teachers' questions 
revolved around content and accessing resources through the network. 



Smplementing an innovation in schools can result 
in practice running before policy. Some existing 
policies need to be "transformed" to match the 
new needs of schools using technology. 

“Our goal should be firsti to understand 
the conditions of pro-social technology use 
and second to employ that understanding for 
learning improvement. Both require more 
penetrating analysis than has heretofore 
been the standard-" 



Dale Mann 
Interactive, Inc. 



Today's classrooms are expected to be technologically up to date. 

The same should be true for the policies that govern technology uses. 
When federal, state, or local district- or building-level policies do not 
keep up with classroom practices, innovative and effective practices 
can grind to a halt. To this end, educators have a leadership role in 
using evaluation information to shape the conversation around the 
kinds of policies that are most supportive in validating best practices 
that enhance the work of the school community. 

Policy issues rise to the surface around data. Who should have access 
to what data in the student information system? In theory, information 
about a student's family situation, for example, can help teachers 
understand and respond to student learning and behavioral problems. 
With today's information technology networks, accessing all kinds of 
personal family information in student files is possible — but what 
are the ethical policy implications for doing so? 

Still another example of how the lack of policies can slow down 
reform has to do with the equitable allocation of computers and other 
technology resources. Does a school distribute computers to students 
who need them the most, or to those students whose teachers show 
the most computer proficiency? What is a school's responsibility 
for out-of-school computer access? How are scarce technology lab 
resources scheduled for use by the school community during and 
after school hours? Does a school have a policy governing the use 
of its technology to address adult technology literacy needs? 
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Many school communities have recognized the need to create and enforce 
Internet usage policies, for instance, but what other less-obvious technology- 
related policies are required to support and govern the best practices associated 
with implementing technology innovations into the school system? Local 
educators have the experience to help shape such questions and define success- 
ful practice for state and federal policymakers. These policymakers can then 
respond by developing policies that support the effective use of technology at 
the local level on a systemic basis. An important part of policy reform is to give 
policymakers a common language and data with which to speak to their con- 
stituents so that support for effective uses of technology will be widespread 
throughout the community. 

[x* Kayenta Unified School District (KUSD) is a small rural Navajo community 
located in an isolated region in the northeastern corner of Arizona, near 
the magnificent Monument Valley. This school district serves 2,600 
students from Kayenta and several other smaller, more rural communities. 
The nearest public library is 100 miles away, while the nearest museums, 
bookstores, and universities are 150 miles. This isolation has provided 
the motivation to use technology to assist in increasing literacy, while 
permitting students to sustain critical elements of the rich traditional life 
of generations of Dine. After ten years of hard work building their 
technology infrastructure, KUSD presently has all six schools and 
administrative buildings, and all classrooms, offices, and administrators 
connected to an Internet/intranet e-mail system. Determining ways to 
complement traditional instruction and community values with the global 
access provided by the Internet has been a compelling policy, as well 
as curricular, challenge faced by Kayenta and other rural isolated school 
districts. Kayenta distance learning policies opened the school community 
to the outside world in many ways. 

More and more as teachers and parents gain access to e-mail 
communications and evaluation data via the Internet, schools are 
finding it difficult to maintain current information policies governing 
information access. A case in point is the use of e-mail for parent-teacher 
communication. While in theory, frequent communications between 
parents and teachers is a positive move forward, practitioners pointed 
out that having to respond to frequent requests from parents about 
their children's schoolwork tears them away from instructional planning 
time. This raises policy questions about teachers' obligations to respond 
to individualized e-mail requests from parents. 

Educators at the conference demonstrated their broadened view of out- 
comes by recounting stories of timely access, attitudinal change, and 
increased motivation. Most initiatives in their early stages feature such 
stories. Their telling is an important step in shaping realistic public and 
legislative expectations for technology evaluation and supportive policies. 
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Schools that have partnered with other schools, universities, 
and educational service agencies to collaborate on technology 
planning, implementation, and research show compelling and 
productive applications of technology. Now comes the call for 
rigorous technology evaluation designs that are innovative and 
relevant to showing its impact and meeting the new national goal 
for educational technology, which states: Research, development, 
and evaluation will shape the next generation of technology 
applications for teaching and learning. 

Researchers and educators are pioneering pathways to evaluating 
the technology initiatives that they've instituted. Such partnerships 
are revolving around many different purposes. Universities are 
partnering with schools to construct the next generation of evaluation 
tools and processes. State and federal governments are beginning 
to reserve grant monies for evaluation activities in order to identify 
and disseminate information about technology practices that work 
and that may benefit schools in other contexts. Policies are beginning 
to be discussed that will support these and other innovative practices. 

While leaders engage in activities to research, develop, and evaluate 
needed methods, schools continue to be under pressure to deliver 
reliable evaluation data about their technology. The Planning for 
Data-Driven Decisions About Technology CD-ROM, included in 
this booklet, can help school improvement teams that are grappling 
with these critical issues. The enclosed CD-ROM contains a database 
of technology and evaluation resources, national conference white 
papers, state technology profiles, research-based tips, and a spotlight 
case example. Insights gleaned from these resources will help school 
improvement teams answer the guiding questions embedded in 
the open-ended action planning tool on the CD-ROM. 
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Planning for D^T CD-ROM 

The ultimate goal for using the enclosed Planning for D^T CD-ROM 
is to generate a comprehensive technology evaluation plan. In the 
process of creating the plan, you can learn more about the critical 
issues involved in technology evaluation and be well prepared to 
address the concerns of your school community and policymakers 
who ask, "What difference does technology make within our school?" 
and "Are our students learning what they need to know to succeed 
in the 21st century?" 

The evaluation planning process embedded on the CD-ROM takes 
a three-fold approach to designing an evaluation for technology in 
education. It asks school improvement teams to think about evaluating 
technology from the systemic organization perspective, from the 
teaching practice perspective, and from the student learning perspective. 
When combined, these perspectives lead to a comprehensive 
technology evaluation plan that your school community can implement. 

Primary Audience: 

• School improvement teams involved in technology planning 

• Administrators responsible for resource allocations 

• Technology and curriculum coordinators involved in 
technology infusion 

Primary Benefits: 

• Align evaluation questions and methods with school 
improvement goals and appropriate standards 

• Develop an understanding of evaluation methods and tools 
applicable to technology 

• Identify useful data analysis and reporting techniques that can 
yield sound data-driven decisions 

• Design a printable action plan for your school community 
to implement 
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System Requirements 

To run Planning for Data-Driven Decisions About Technology on a PC-compatible 
computer, your equipment must meet or exceed the following system requirements: 

• Processor — Pentium 166 or better; Windows 98 

• Memory — at least 32 MB 

• CD-ROM Drive — 4x or better 

• Sound— sound card and speakers (required for multimedia clips) 

To run Planning for Data-Driven Decisions About Technology on a Macintosh 
computer, your equipment must meet or exceed the following system requirements: 

• Processor — PowerMac/PC or better; System 8.0 or later 

• Memory — at least 32 MB free after system loads 

• CD-ROM Drive — 4x or better 

• Sound — sound card and speakers (required for multimedia clips) 

To start the program, double-click your CD-ROM drive labeled "D3T," 
then double-click the "D3T" icon. See the "Readme.text" file for more details. 



Planning for DPT CD-ROM 

The ultimate goal for using the enclosed Planning for fpT CD-ROM 



is to generate a comprehensive technology evaluation plan. In the 
process of creating the plan,' you can learn more about the critical 
issues involved in technology evaluation and be well prepared to 
address the concerns of your school community and policymakers 
who ask, "What difference does technology make within our school?" 
and "Are our students learning what they need to know to succeed 
in the 21st century?" 

The evaluation planning process embedded on the CD-ROM takes 
a three-fold approach to designing an evaluation for technology in 
education. It asks school improvement teams to think about evaluating 
technology from the systemic organization perspective, from the 
teaching practice perspective, and from the student learning perspective. 
When combined, these perspectives lead to a comprehensive 
technology evaluation plan that your school community can implement. 

|> Primary Audience: 



• School improvement teams involved in technology planning 

• Administrators responsible for resource allocations 

• Technology and curriculum coordinators involved in 
technology infusion 



Primary Benefits: 

• Align evaluation questions and methods with school 
improvement goals and appropriate standards 

• Develop an understanding of evaluation methods and tools 
applicable to technology 

• Identify useful data analysis and reporting techniques that can 
yield sound data-driven decisions 

• Design a printable action plan for your school community 
to implement 
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The Secretary's Conference on Educational Technology- 1999 



Critical Issues in Evaluating the Effectiveness of Technology 

by 

Mary McNabb, North Central Regional Educational Laboratory 
Mark Hawkes, Dakota State University 
Ullik Rouk, Policy Studies Associates 

We are far enough along in the technological revolution and its application to learning that it 
is time for systematic review and analysis of what works best. 



U.S. Secretary of Education Richard W. Riley 
National Conference on Educational 
Technology 

Washington, D.C., July 12, 1999 

The Secretary's Conference on Educational Technology: Evaluating the Effectiveness of Technology on 
July 12-13, 1999, in Washington, D.C., noted a shift in schools' focus on technology. Where once the 
emphasis was on’ building and implementing a technology infrastructure, today it is on evaluating the 
effectiveness of its use in schools and classrooms. Parents and teachers, school boards and 
administrators, governors and state legislatures, and Congress all want to know if the nation's investment 
in technology is providing a return in student achievement. Indeed, if resources are to be expended on 
technology, it is becoming a political, economic, and public policy necessity to demonstrate its vital 

effectiveness. 

In his opening address, U.S. Secretary of Education Richard Riley remarked, "The primary reason for 
this conference is to gather information from all of the outstanding schools, districts, and states 
represented here-so that we can study it, share it, and learn from it. Just as important as learning what 
works, we must learn what does not work. We must not assume everything that employs technology is 
going to be successful. That is why evaluation is so important. And then we must use that evaluation to 

create positive change." 

The conference drew on the insights and collective wisdom of its attendees, starting with an emphasis on 
state-level technology evaluations. Evaluators from West Virginia explained how they isolated the 
effects of their Basic Skills/Computer Education initiative. They found that the more access to 
technology students had and the more their teachers believed that technology could help and were 
trained to use the technology, the higher students scored on the Stanford 9 (1 1% of the total gam scores). 
Idaho attendees described a four-year study focused on eight specific goals by which to evaluate the 
impact of the state's technology investment. Significant results included statewide academic gains as 

measured by the Iowa Test of Basic Skills (ITBS) and the Test of Academic Proficiency (TAP) for 8 th 
and 11 th graders. 

From there, the conference turned its spotlight on school practitioners who expressed the need for more 
formative evaluations than the summative evaluations described by state policymakers. Lively debates 
arose among teachers, district curriculum and technology coordinators, administrators, state curriculum 
and technology coordinators, state and national policymakers, and researchers in the evaluation of 
educational technology about ways of identifying and collecting technology evaluation data that is 
relevant at the local level but also useful for other stakeholders. 
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